Skip to content

feat(model): add model support verification cards#4909

Draft
yaoyu-33 wants to merge 87 commits into
mainfrom
yuya/mb-575-model-support-cards
Draft

feat(model): add model support verification cards#4909
yaoyu-33 wants to merge 87 commits into
mainfrom
yuya/mb-575-model-support-cards

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds concise, agent-readable model support cards for four variants:

  • Qwen/Qwen3-8B
  • Qwen/Qwen3-30B-A3B
  • moonshotai/Moonlight-16B-A3B
  • nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

It also adds the repo-local create-model-verification-card skill, its deterministic validator, and a helper for exact-length, byte-stable HF inference verification.

The cards distinguish bounded convergence evidence from benchmark-only performance evidence, pin immutable Hugging Face and dataset revisions, preserve recipe-owned GBS/MBS/LR contracts, require saved post-setup runtime configuration, and keep private executor wiring out of tracked cards and PR text.

Card contract

  • Keep all twelve core items; include pretrain_performance only when the exact variant has a canonical public performance recipe.
  • Use portable public launchers and short logical work/... paths.
  • Pin immutable model, tokenizer, and dataset revisions plus split, seeds, and packing alignment.
  • Keep recipe GBS/MBS/LR unchanged in verification commands.
  • Record initial/final loss plus final-ten mean step time and model TFLOP/s/GPU for every verified training item.
  • Require exact-length, byte-stable inference output from two executions.
  • Verify manual forward correlation with an exact next-token match and cosine similarity at least 0.99; maximum and mean absolute logit differences remain report-only diagnostics.
  • Resume directly from the uninterrupted bounded-pretrain step-50 checkpoint to step 100 in a distinct output root.
  • Treat submission or a short smoke as evidence-in-progress, not verification.

Current verification status

The branch remains a draft while the remaining clean-contract workloads finish. Current head: 5ec17e4.

Variant Verified Remaining
Qwen3-8B 10/12 Bounded pretrain and direct checkpoint resume
Qwen3-30B-A3B 10/13 Full SFT, post-SFT export/inference, and PEFT
Moonlight-16B-A3B 7/12 Bounded pretrain/resume, full SFT/export, and PEFT
Nemotron 3 Nano 4B 12/12 None

The three long-context items are now verified from one coherent clean-run evidence cohort:

Variant Loss Final-ten step time Final-ten TFLOP/s/GPU Packing Actual supervised tokens
Qwen3-8B 1.611622 → 1.490618 71668.780 ms 34.080 99.97% 3,444,917
Qwen3-30B-A3B 1.645009 → 1.468103 142663.710 ms 26.120 99.28% 13,573,663
Moonlight-16B-A3B 1.306440 → 1.228019 60599.310 ms 40.840 99.69% 13,765,732

Each completed exactly 20 optimizer steps with finite losses, zero skipped/NaN iterations, a persisted runtime config, and independently reconstructed packed-sample accounting.

Revision hardening and validation

  • Hugging Face model and tokenizer revisions are persisted by the affected recipes instead of relying on mutable defaults.
  • The revision-hardening patch passed 184 targeted tests and repository-wide pre-commit.
  • The current long-context/card update passed all three card validators, tokenizer unit tests (9 passed), Ruff checks, the CI-equivalent secrets scan, and repository-wide pre-commit.
  • Two public immutable revision literals triggered the generic high-entropy detector; narrow inline allowlist annotations now scope those known false positives to the exact literals.
  • git diff --check passes, and an independent evidence review reproduced every promoted metric and supervised-token count.

Remaining work

Fresh revision-pinned bounded runs are progressing through an independently reviewed dependency DAG. The two full-SFT sources have completed their 100-step workloads and produced final checkpoints, while bounded pretrain, PEFT, direct-resume, and post-SFT export/inference paths continue. Execution state alone does not promote an item: each remains unverified until its exact optimizer-step set, finite-loss/no-skip gates, saved runtime config, checkpoint or adapter artifacts, packing/supervised-token evidence, resume sentinels, and deterministic export inference all pass.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 added docs-only With great power comes great responsibility. docs Documentation-only updates or documentation debt area:model Model implementations and HF bridge logic labels Jul 16, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title docs(model): draft model support card with Qwen3 example docs(model): add minimal Qwen3 support card draft Jul 16, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Comment thread model_cards/qwen3-8b/card.yaml
Comment thread model_cards/qwen3-8b/card.yaml
Comment thread model_cards/qwen3-8b/card.yaml
yaoyu-33 added 8 commits July 16, 2026 15:51
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title docs(model): add minimal Qwen3 support card draft docs(model): add verified Qwen3 support card Jul 17, 2026
yaoyu-33 added 4 commits July 16, 2026 18:29
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…port-cards

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title docs(model): add verified Qwen3 support card docs(model): add Qwen3 support card Jul 17, 2026
yaoyu-33 added 5 commits July 16, 2026 22:29
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 changed the title docs(model): add Qwen3 support card docs(model): add Qwen3 support card and verification skill Jul 17, 2026
Comment thread model_cards/qwen3-8b/card.yaml Outdated
yaoyu-33 added 2 commits July 19, 2026 22:18
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33 yaoyu-33 added feature New capabilities, enhancements, or enablement work and removed docs-only With great power comes great responsibility. docs Documentation-only updates or documentation debt labels Jul 20, 2026
yaoyu-33 added 13 commits July 19, 2026 23:36
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Base automatically changed from yuya/qwen3-verification-prerequisites to main July 20, 2026 20:24
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

Nemotron 3 Nano 4B verification is complete at card commit f04e941906e05d4ef69c4c1e5fc7b3b78e12c844. Runtime evidence is pinned to clean Bridge commit 0e6c1837edeb442b07c9175b855b3d288c5cad33, MCore 61f31145dfb4cfaa7d285df12aa9abffb22ff485, and immutable HF revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f.

Core item Status Result
HF → Megatron, CPU verified Reloadable checkpoint; paired round trip exact
HF → Megatron, GPU verified Reloadable checkpoint; paired round trip exact
Megatron → HF, CPU verified 263/263 tensors and 3,973,556,832 parameters bitwise exact; native reload
Megatron → HF, GPU verified Same exact audit; generation config byte-exact; all 21 Mamba A_log/D/dt_bias/out_proj groups exact
Manual HF/Megatron parity verified Same next token 6993 ( Paris); cosine 0.999982; max/mean logit difference 0.093750/0.018522
Deterministic Megatron inference twice verified Exactly 2 tokens twice, identical completion Paris.
Bounded random-init pretraining verified 100/100 steps; loss 12.432500 → 5.511951; last-10 26,775.11 ms and 402.51 TFLOPS/GPU; finite, no skipped/NaN iterations; midpoint/final artifacts and final reload verified
Full SFT verified 100/100 steps; loss 1.839282 → 1.234906; last-10 627.0 ms and 262.89 TFLOPS/GPU; 99.27% packed; final checkpoint reload verified
Final SFT export + HF inference twice verified BF16 export with 263 weights; native reload; exactly 45 identical generated tokens twice
32K packed SFT with CP=2 verified TP2/CP2/SP, 100/100 steps; loss 1.658660 → 1.100197; last-10 5,575.59 ms and 142.06 TFLOPS/GPU; 99.35% packed; final reload verified
PEFT verified LoRA 100/100 steps; loss 1.769892 → 1.256916; last-10 381.69 ms and 431.90 TFLOPS/GPU; 99.36% packed; adapter reload verified
Direct midpoint resume verified Restored step 50 and ran exactly 51–100; step 51 matched exactly; step-100 loss 5.512038 vs 5.511951 reference (0.00158%); final checkpoint reload verified

The card is at model_cards/nemotron-3-nano-4b/card.yaml. The exact 4B variant has no canonical public performance recipe, so pretrain_performance is intentionally omitted.

Sanitized fixes covered immutable HF-revision plumbing, strict generation-config preservation, native Transformers loading for the fixed Nemotron-H implementation, Mamba chunk-size mapping, real Megatron inference context/InferenceMode with TP-aware output handling, and opt-in synchronous training via --wait. The first TP1/CP2 32K attempt exceeded memory; the semantics-preserving TP2/CP2/SP topology completed all 100 steps. Continual pretraining from the released HF checkpoint and the damaged/rare embedding-row investigation remain out of scope in #4967.

Final clean gates: model-card validator plus privacy deny terms passed; targeted secret scan passed; 136 focused tests passed; git diff --check passed; full pre-commit passed.

yaoyu-33 added 9 commits July 20, 2026 15:58
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic feature New capabilities, enhancements, or enablement work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants